fix: fail fast on endpoint response stalls - #462

Summary

• add an opt-in no-progress deadline for in-flight endpoint requests
• expose it as settings.timeouts.no_progress_timeout_s plus a --no-progress-timeout CLI alias
• arm the guard only while work is in flight; retire it as soon as the phase drains

When this is useful

Use this for automated runs where a request can be accepted but the endpoint then becomes silent — for example, a TensorRT-LLM disaggregated executor or KV-transfer stall that never reaches the normal terminal-error path. It is engine-agnostic: it catches the same client-visible silent failure through vLLM, a frontend, or transport. Without it the benchmark blocks until its outer wall-time limit, because the phase drain waits on responses that never arrive and drain_timeout defaults to unlimited.

Disabled by default. It starts after work is issued, runs only while requests are in flight, and resets on an observed stream chunk or final result. It does not diagnose or restart the backend — it makes the benchmark fail with a clear error. For non-streaming endpoints, configure it above the full expected request latency. For TensorRT-LLM disaggregated serving the documented starting value is 300 s, matching the executor hang_detection_timeout.

Changes since the last review

Rebased onto latest main and moved the setting into the Timeouts model added by #409 (@viraatc).

Measured cost when disabled (A/B against the base commit): +3.5 ns per response, +9.4 ns per request. With stream_all_chunks: false (the default) the main process sees 2 messages per request, not one per token.

Measured cost of the arming change under the production eager task factory: arm+cancel per request went from 2205 ns to 535 ns (4.1x).

Also fixed while here

A receiver transport error could overwrite an earlier NoProgressError. A stalled endpoint often drops its connection too, so the real diagnosis was being masked by a generic "receiver failed" message. First error now wins, with a regression test.

Not adopted

The suggestion to use one phase/session-lifetime watchdog driven by an activity event. The guard is still armed on the 0→1 in-flight transition and retired on drain — now with a TimerHandle rather than a task. This keeps the timer heap clear once a phase drains, and measured 4.1x cheaper than the reviewed version. Happy to switch to a resident watchdog if that is preferred.

Validation

• pytest tests/unit/config/test_schema.py tests/unit/commands/test_benchmark.py tests/unit/load_generator/test_async_session.py — 425 passed
• pre-commit run --all-files — all hooks pass except mypy, which reports 3 pre-existing os.sched_*affinity errors on macOS in cpu_affinity.py / token_metrics.py; neither file is touched by this PR and the symbols exist on Linux
• python scripts/regenerate_templates.py --check
• AGA disaggregated held-response canary, job 578946: failed as intended with Endpoint made no response progress for 10.0s with 1 request(s) in flight, 10.002 s after the phase started against a 10 s deadline. The 10 s value is fault-injection coverage, not the deployment recommendation.

Timer lifecycle is covered in both directions: the perf issue cap (stop_current_phase) deliberately leaves the guard armed so a stall during drain is still caught, while drain completion, stop(), a phase change, and session teardown each retire it.